If evidence is to influence decisions on global development policy, it is vital that it is reliable and robust. Although randomised control trials (RCTs) represent the gold standard for studying causal relationships, contextual and financial constraints may preclude the undertaking of high quality RCTs in many circumstances. However, RCTs do not necessarily have to be more costly than qualitative studies; the expensive part is often the household survey, not the randomisation itself. If conducted using administrative data, RCTs can be relatively cheap. Nevertheless, capacity issues often mean that there is a mismatch between the desire to implement an RCT and an understanding of what is required. RCTs were seen to be particularly valuable for high-stakes, strategic funding decisions, not as a means to replace every evaluation. Strong other quantitative and qualitative methods were also mentioned as alternatives to standard evaluations. Meanwhile a philosophical divide is often perceived between ‘researchers’ and ‘evaluators’, preventing evidence collected using a variety of methods from being joined up to deliver a common message. Further, although data produced by RCTs and other evaluative methods is necessarily rooted in the past, in an increasingly unstable world it may have limited utility in preparing policymakers to face future challenges
“We should create a more versatile toolbox of methods.”
Selecting appropriate methods. Participants highlighted the importance of ensuring that evidence producers employ the right approach to tackling specific questions. While RCTs represent a valuable tool, evidence can and should be comprised of data derived from studies employing a variety of methods and it is vital that researchers and evaluators are supported in choosing methods appropriate to the issue they are attempting to understand. It is also crucial that evidence producers are assisted in selecting methods appropriate to different points in the research cycle, maximising evidence utility and enhancing the likelihood of uptake.
Utilising creative methodological approaches. The value of evidence based on methods other than RCTs was also emphasised at the local level. The feasibility of employing traditional methods in conflict environments, for example, was seen as necessitating a creative approach to evidence generation that might rely on analysing existing data, while indigenous perspectives and methodologies were positioned as having the potential to shed new light on context-specific issues and increase the likelihood of interventions tackling the correct binding constraint. Meanwhile, more collaborative and participatory methods were seen as increasing the likelihood of evidence uptake.
Facilitating methodological diversity. Several strategies were put forward to encourage the diversification of research methods in evidence production. Helping researchers and evaluators in understanding the various research available to them was highlighted as a priority. Considering a range of potential methods at the commissioning stage rather than selecting a method on the basis of its reputation or the random distribution of expertise was also seen as an important step. Participants were also encouraged to be aware of their own limitations and biases when commissioning and undertaking research, and to be willing to take into account alternative perspectives on evidence generation. Meanwhile, bridging the tensions and differences between researchers and evaluators has the potential to bring together different methodological approaches behind a shared consensus that evidence is worth fighting for.